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The  Effects  of  Previous  Computer  Experience  on  Air  Traffic- 
Selection  and  Training  (AT-SAT)  Test  Performance 


Computers  have  been  used  more  frequently  in  test 
administration  in  recent  years.  Many  tests  that  have 
traditionally  been  administered  in  paper-and-pencil 
format  are  now  administered  on  a  computer  worksta¬ 
tion  (Finegan  &  Allen,  1994;  Lee,  1986;  Mead  & 
Drasgow,  1993).  Computers  provide  a  faster  and 
easier  method  of  assessing  performance  and  cognitive 
ability.  Additionally,  computers  provide  a  means  to 
measure  performance  using  dynamically  changing 
stimuli  or  scenarios  (Mead  &  Drasgow,  1993).  One 
disadvantage  of  computer  administered  tests,  how¬ 
ever,  is  the  inadvertent  measurement  of  extraneous 
abilities  related  to  prior  experience  with  a  computer 
keyboard  or  mouse,  rather  than  measurement  of 
relevant  knowledge,  skills,  abilities,  and  other  char¬ 
acteristics  (KSAOs)  needed  for  the  job.  This  issue  is 
particularly  important  in  the  realm  of  personnel  selec¬ 
tion,  given  the  legal  ramifications  and  potential  adverse 
impact  that  may  result  from  incorporating  information 
into  the  selection  process  that  is  not  job-related.  The 
purpose  of  this  paper  is  to  explore  the  relationship 
between  prior  computer  experience  and  performance  on 
a  computerized  personnel  selection  test. 

Although  the  use  of  computers  in  the  selection 
process  is  becoming  increasingly  popular,  there  has 
been  relatively  little  written  in  the  scientific  literature 
about  their  use  by  organizations.  The  effect  of  com¬ 
puter  experience  on  computerized  test  performance 
has  received  more  attention  in  the  education  litera¬ 
ture,  where  researchers  have  studied  two  primary 
issues  related  to  computerized  testing:  computer  anxi¬ 
ety  (Bradley  &C  Russell,  1997;  Dimock  &C  Cormier, 
1991;  Legg  &  Buhr,  1992;  Levine,  &  Donista- 
Schmidt,  1998;  Powers  &  O’Neill,  1993)  and  the 
equivalence  of  paper-and-pencil  and  computerized 
tests  (Finegan  &  Allen,  1994;  Mead  &  Drasgow, 
1993).  Two  findings  have  generally  emerged:  1) 
people  with  more  computer  experience  reported  less 
computer  anxiety,  while  people  with  higher  levels  of 
computer  anxiety  failed  to  perform  as  well  on  com¬ 
puterized  tests,  and  2)  the  differences  in  performance 
between  paper-and-pencil  and  computerized  tests 
were  negligible. 


The  current  study  examines  the  relationship  be¬ 
tween  computer  experience  and  test  performance 
using  a  computerized  selection  test.  One  of  the  main 
objectives  of  this  study  was  to  determine  if  examinees 
with  more  computer  experience  perform  better  than 
examinees  with  less  computer  experience.  More  spe¬ 
cifically,  does  previous  computer  experience  give 
examinees  an  advantage  when  taking  a  computerized 
test?  The  converse  is  of  particular  importance  in  the 
arena  of  personnel  selection:  Does  a  lack  of  computer 
experience  put  examinees  at  a  disadvantage?  In  a 
study  exploring  the  performance  of  undergraduate 
students  on  a  computerized  test  of  arithmetic  reason¬ 
ing,  Lee  (1986)  found  that  prior  computer  experience 
improved  performance.  She  observed  that  computer¬ 
ized  testing  might  discriminate  against  those  who 
have  not  worked  with  computers.  This  should  be  of 
great  importance  to  hiring  organizations  since  all 
racial  groups  do  not  have  the  same  access  to  comput¬ 
ers.  According  to  a  survey  conducted  by  the  U.S. 
Department  of  Commerce  (1999),  46.6%  of  White 
households  have  a  computer,  as  compared  to  only 
23.2%  of  Black  households  and  25.5%  of  Hispanic 
households.  Further  analyses  revealed  that  white 
households  have  more  computers  than  other  racial 
groups  at  all  levels  of  income  (U.S.  Department  of 
Commerce,  1999). 

Another  objective  of  the  current  study  was  to 
determine  if  the  relationship  between  computer  ex¬ 
perience  and  computerized  test  performance  remains 
consistent,  regardless  of  the  type  of  test  being  pre¬ 
sented  in  the  computerized  format.  For  example 
some  computerized  tests,  often  referred  to  as  “page- 
turner”  tests,  present  items  that  have  been  adapted  to 
the  computer  but  could  also  be  presented  via  another 
medium  such  as  paper-and-pencil.  In  most  cases,  the 
examinee  uses  the  keyboard  to  select  a  response  to  the 
questions,  which  are  presented  one  at  a  time.  Another 
type  of  test  is  that  which  takes  the  form  of  dynamic 
virtual  scenarios,  work  samples,  or  simulations.  The 
dynamic,  scenario-based  items,  which  may  resemble 
a  video  or  computer  game,  often  require  extensive  use 
of  both  a  mouse  and  a  keyboard.  An  examinee  with 
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no  computer  experience  may  be  at  a  disadvantage 
when  responding  to  items  that  require  these  com¬ 
puter  skills.  This  proposal  would  be  consistent  with 
the  findings  of  Young,  Broach,  &  Farmer  (1997), 
who  observed  that  self-reported  video  game  experi¬ 
ence  was  related  to  performance  on  a  computer-based 
air  traffic  scenario  test. 

The  literature  review  and  summary  of  relevant 
issues  presented  above  led  to  the  following  hypoth¬ 
esis:  A  relationship  exists  between  computer  experi¬ 
ence  and  performance  on  computerized  tests; 
specifically,  people  with  prior  computer  experience 
will  earn  higher  scores  on  the  computerized  selection 
tests.  Furthermore,  the  relationship  between  com¬ 
puter  experience  and  test  performance  will  be  greater 
on  tests  that  require  the  use  of  a  mouse  in  addition  to 
the  keyboard,  such  as  in  the  case  of  dynamic,  sce¬ 
nario-based  tests. 

METHOD 

Participants 

In  an  effort  to  ensure  that  the  participants  in  this 
study  were  representative  of  potential  applicants, 
they  were  required  to  meet  the  basic  age  and  educa¬ 
tional  qualifications  for  the  air  traffic  controller  job. 
A  total  of  96  participated  in  the  study,  55  male  and 
41  female,  between  the  ages  of  18  and  30:  Other 
demographic  information  is  presented  in  Table  1 .  An 
outside  personnel  agency  recruited  participants,  who 
were  paid  an  hourly  rate. 

Measures 

Computer  Experience.  Computer  experience  was 
measured  using  the  Computer  Use  and  Experience 
Questionnaire  (CUE)  developed  by  Potosky  and 
Bobko  (1997).  This  questionnaire  consisted  of  twelve 
items  (Appendix  A)  that  are  answered  on  a  5-point 
Likert  Scale  (1  =Strongly  Disagree,  2=Disagree,  3=Nei- 
ther  Agree  nor  Disagree,  4=Agree,  5=Strongly  Agree) . 
The  purpose  of  administering  the  questionnaire  was 
to  determine  the  subject's  knowledge  of  computers 
and  to  assess  the  extent  to  which  he  or  she  has  used  a 
computer.  These  items  were  summed  to  create  a 
composite  score.  Potosky  and  Bobko  reported  a  coef¬ 
ficient  alpha  of  .92. 

For  the  purpose  of  analyses,  the  composite  com¬ 
puter  experience  score  was  used  as  the  measure  of 
computer  experience  for  all  analyses  other  than  the 
Multiple  Analysis  of  Variance  (MANOVA).  The 


Table  1.  Demographics 


Variable 

Number 

Percent 

Age 

18-20  years 

20 

20.8 

21-23  years 

26 

27.1 

24-26  years 

23 

23.9 

27-30  years 

27 

28.1 

Gender 

Male 

55 

57.3 

Female 

41 

42.7 

Race 

Minority 

36 

39.6 

Non-Minority 

55 

60.4 

Education 

High  School 

24 

25.3 

Trade  School 

8 

8.4 

Attend  College 

49 

51.6 

College  Degree 

10 

10.5 

Graduate  School 

4 

4.2 

Table  2.  Computer  Experience  Items 


I  am  computer  literate. 

•  Agree 

60.4% 

•  Disagree 

39.6% 

I  regularly  use  a  PC  for 
word  processing. 

•  Agree 

57.3% 

•  Disagree 

42.7% 

I  am  good  at  using 
computers. 

•  Agree 

52.1% 

•  Disagree 

47.9% 
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item  “I  regularly  use  a  PC  for  word  processing”  was 
used  individually  in  the  MANOVA  to  improve  un¬ 
derstanding  of  the  relationship  between  the  use  of  a 
computer  for  these  specific  tasks  and  performance  on 
computerized  tests.  The  distribution  of  responses  to 
three  specific  items,  “I  am  computer  literate,”  “I 
regularly  use  a  PC  for  Word  Processing,”  and  “I  am 
good  at  using  computers,”  is  shown  in  Table  2. 

Personnel  Selection  Test.  The  personnel  selection 
test  used  in  the  current  study  is  the  Air  Traffic- 
Selection  and  Training  (AT-SAT)  test,  a  newly  devel¬ 
oped,  computerized  test  of  cognitive  ability.  This  test 
will  be  used  by  the  FAA  to  select  individuals  into  the 
Air  Traffic  Control  Specialist  (ATCS)  job.  The  AT- 
SAT  battery  is  comprised  of  seven  tests  of  cognitive 
ability  and  one  non-cognitive  measure.  The  non- 
cognitive  measure  was  included  in  the  AT-SAT  com¬ 
posite  score  but  was  not  used  independently  in  the 
current  study,  as  the  focus  of  the  study  was  on 
cognitive  ability  tests.  In  addition  to  the  AT-SAT 
composite  score  based  on  total  test  performance, 
scores  were  also  calculated  for  each  cognitive  test 
included  in  the  battery.  A  description  of  each  subtest 
follows. 

The  Applied  Math  test  contains  30  multiple-choice 
questions.  The  first  five  items  are  practice  questions 
followed  by  25  scored  items.  An  example  of  an 
Applied  Math  question  is:  A  plane  has  flown  for  3 
hours  with  a  ground  speed  of 21 0  knots.  How  far  did  the 
plane  travel ?  Each  of  these  questions  requires  the 
subject  to  make  calculations  based  on  time,  speed, 
and  distance  to  identify  the  correct  answer  from 
among  four  choices. 

The  Angles  test  measures  the  subject’s  ability  to 
recognize  the  measurement  of  angles.  This  test  con¬ 
tains  30  multiple-choice  questions  with  four  response 
options.  There  are  two  types  of  questions  on  the  test. 
The  first  type  presents  a  picture  of  an  angle  and 
requires  the  subject  to  estimate  (in  degrees)  the  cor¬ 
rect  size  of  the  angle.  The  second  type  presents  a 
measure  in  degrees  (e.g.  “35°”)  and  asks  the  subject  to 
choose  the  depicted  angle  that  best  represents  that 
degree  measurement. 

The  Letter  Factory  test  (LF)  simulates  four  factory 
assembly  lines,  each  of  which  manufactures  one  of 
four  letters  of  the  alphabet  (A,  B,  C,  or  D)  in  one  of 
three  colors.  The  test  requires  that  subjects  use  a 
mouse  to  perform  multiple  and  often  concurrent 
tasks.  Each  test  section  begins  with  letters  that  appear 
at  the  top  of  the  conveyor  belts  and  move  down 


toward  the  loading  area.  The  object  of  the  test  is  to 
“load”  each  of  the  colored  letters  into  boxes  that 
correspond  to  the  letter’s  color  (e.g.,  an  orange  letter 
must  go  into  an  orange  box).  Based  on  the  letters  on 
the  conveyor  belts,  subjects  immediately  begin  se¬ 
lecting  and  moving  boxes  to  the  loading  area  to 
provide  just  the  right  number  and  color  of  boxes  to 
correctly  place  all  letters.  Other  tasks  performed 
during  the  simulated  factory  settings  include:  (1) 
ordering  new  boxes  when  supplies  become  low,  and 
(2)  calling  Quality  Control  when  defective  letters 
appear  (i.e.,  letters  that  are  not  As,  Bs,  Cs,  orDs).The 
LF  test  produces  two  scores:  LF  situational  awareness 
and  LF  planning  and  thinking  ahead. 

The  Air  Traffic  Scenarios  Test  (ATST)  is  a  low- 
fidelity  simulation  of  an  air  traffic  control  (ATC) 
radar  screen  that  is  updated  every  seven  seconds.  The 
goal  is  to  maintain,  as  efficiently  as  possible,  separa¬ 
tion  and  control  of  a  varying  number  of  simulated 
aircraft  (represented  as  data  blocks)  within  the  desig¬ 
nated  airspace.  Aircraft  in  flight  can  pass  through  the 
airspace  or  land  at  one  of  two  airports  within  the 
airspace.  Each  aircraft’s  data  block  indicates  its  present 
heading,  speed,  and  altitude.  There  are  eight  differ¬ 
ent  headings  representing  45  degree  increments,  three 
different  speeds  (slow,  moderate,  fast),  and  four  dif¬ 
ferent  altitude  levels  (1  =lowest  and  4=highest).  Sepa¬ 
ration  and  control  are  achieved  by  communicating 
and  coordinating  with  each  aircraft  by  using  the 
computer  mouse  to  click  on  the  data  block  represent¬ 
ing  each  aircraft  and  providing  instructions  such  as 
changes  to  the  current  heading,  speed,  or  altitude. 
The  ATST  produces  three  scores:  AT  Efficiency,  AT 
Safety,  and  AT  Procedural  Accuracy. 

In  the  Scan  test,  subjects  monitor  a  field  on  the 
screen  that  contains  discrete  objects  (called  data 
blocks),  which  are  Vi-inch  tall  and  are  moving  in 
different  directions.  Data  blocks  appear  in  the  field, 
travel  in  a  straight  line  for  a  short  period  of  time,  and 
then  disappear.  During  the  test,  the  subject  sees  a 
blue  field  that  fills  the  screen,  with  the  exception  of 
a  two-inch  white  bar  at  the  bottom.  In  this  field,  up 
to  12  green  data  blocks  may  be  present.  Each  data 
block  contains  two  lines  of  letters  and  numbers 
separated  by  a  horizontal  line.  The  upper  line  is  the 
identifier  and  begins  with  a  letter  followed  by  a  two- 
digit  number.  The  lower  line  contains  a  three-digit 
number.  Subjects  are  scored  on  the  speed  with  which 
they  notice  and  respond  to  the  data  blocks  that  have 
a  number  on  the  lower  line  outside  a  specified  range. 
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Throughout  the  test,  this  range  is  displayed  at  the 
bottom  of  the  screen  (e.g.,  360-710).  To  “respond” 
to  a  data  block,  the  subject  types  the  two-digit  num¬ 
ber  from  the  upper  line  of  the  block  (ignoring  the 
letter  that  precedes  it),  and  then  presses  “enter.” 

The  Dial  Reading  test  measures  the  subject’s  abil¬ 
ity  to  quickly  identify  and  accurately  read  certain 
dials  on  an  instrument  panel.  Subjects  are  asked  to 
choose  from  one  of  five  response  alternatives  for  each 
question  about  a  given  display.  The  test  consists  of  20 
questions.  Individuals  pace  themselves  against  the 
display  of  time  remaining  in  the  subtest.  Subjects  are 
advised  to  skip  difficult  items  and  return  to  them  at 
the  end  of  the  test.  Each  panel  consists  of  seven  dials 
in  two  rows,  a  layout  that  remains  constant  through¬ 
out  the  test.  Each  of  the  seven  dials  contains  unique 
flight  information  (e.g.,  airspeed,  fuel,  temperature). 

The  Analogies  test  measures  the  subject’s  ability  to 
apply  the  correct  rules  to  solve  a  given  problem  as  well 
as  their  efficiency  in  using  the  available  information 
to  solve  that  problem.  Analogies  are  based  on  words, 
pictures,  or  figures  and  appear  in  three  “windows”  on 
the  same  screen  for  a  given  item.  Subjects  use  a  mouse 
to  move  freely  between  the  three  windows,  view  the 
different  parts  of  the  analogy,  and  select  their  answer. 
However,  they  can  view  only  one  window  at  a  time. 
Window  A  presents  the  first  part  of  the  analogy, 
which  requires  subjects  to  infer  the  underlying  rule. 
Window  B  contains  the  second  part  of  the  analogy, 
which  requires  subjects  to  apply  the  inferred  rule. 
Finally,  Window  C  provides  subjects  the  opportu¬ 
nity  to  confirm  their  choice  by  selecting  their  answer 
from  the  available  response  options.  The  test  has  57 
items:  30  word  analogies  and  27  visual  (i.e.,  either 
pictorial  or  figural)  analogies. 

As  stated  above,  one  of  the  objectives  of  the  cur¬ 
rent  study  was  to  determine  if  performance  on  certain 
types  of  tests  was  affected  more  by  computer  experi¬ 
ence  than  performance  on  other  types.  The  ATST 
and  LF  tests  are  dynamic  scenario-based  tests  that 
require  the  use  of  the  mouse.  The  scan  test  is  also  a 
dynamic  test  but  requires  use  of  the  keyboard  number 
pad,  rather  than  a  mouse.  The  analogies  test  is  a  static 
test  that  requires  use  of  the  mouse  to  view  different 
parts  of  the  screen  and  to  select  the  correct  response. 
The  applied  math,  angles,  and  dial  reading  tests  are 
static  “page-turner”  tests  that  require  only  the  use  of 
the  keyboard  to  select  the  correct  response  for  mul¬ 
tiple  choice  questions. 


Procedure 

All  subjects  participated  in  a  pre-screening  session 
prior  to  administration  of  the  selection  test.  During 
this  time,  they  were  informed  of  the  purpose  of  the 
study,  were  given  voluntary  consent  forms,  and  com¬ 
pleted  biographical  information  questionnaires.  Ten 
participants  could  be  tested  simultaneously  based  on 
the  availability  of  computer  workstations.  Partici¬ 
pants  were  randomly  assigned  to  groups  of  ten.  All 
computer  workstations  were  separated  by  partitions. 
Participants  were  given  the  same  instructions  before 
beginning  the  test  battery.  The  computer  experience 
questionnaire  was  administered  at  the  beginning  of 
the  session  as  part  of  the  AT-SAT  test.  Participants 
received  ten-minute  breaks  between  certain  sections 
of  the  test,  as  well  as  a  45 -minute  break  for  lunch.  The 
average  testing  time  was  approximately  six  hours. 

RESULTS 

The  relationship  between  computer  experience 
and  performance  on  the  computerized  selection  test 
was  investigated  using  Pearson’s  product-moment 
correlations  and  hierarchical  multiple  regression. 
MANOVA  and  t-tests  were  used  to  identify  group 
differences  on  the  dependent  variables.  The  results  of 
these  analyses  are  presented  below. 

Computer  experience/  test  score  relationship 

The  AT-SAT  subtest  correlation  matrix,  presented 
in  Table  3,  revealed  statistically  significant  correla¬ 
tions  between  most  of  the  tests  within  the  AT-SAT 
battery.  Pearson’s  product-moment  correlations  were 
used  to  investigate  the  linear  relationship  between 
computer  experience  and  AT-SAT  scores.  As  shown 
in  Table  4,  there  was  a  moderate  correlation  between 
computer  experience  and  AT-SAT  composite  score 
(r=.38,  p<.01).  An  examination  of  relationships  be¬ 
tween  computer  experience  and  individual  AT-SAT 
subtest  scores  revealed  that  computer  experience  was 
not  significantly  correlated  with  the  dial  reading, 
scan,  and  AT-Scenarios  procedural  accuracy  scores. 
Computer  experience  was  most  highly  correlated 
with  the  letter  factory  (LF)  situational  awareness 
(r=.45,  £<.01),  applied  math  (r=.37,  p<.01),  and  AT 
efficiency  (r=.365,  p<.0 1)  measures.  As  demonstrated 
by  these  correlations,  and  contrary  to  the  hypothesis. 


4 


Table  3.  Correlation  of  AT-SAT  Sub-tests 


1 

2 

3 

4 

5 

6 

7 

8 

9 

10 

ii 

12 

1.  Dials 

1.00 

2.  Angles 

.603* 

1.00 

3.  Amath 

.559* 

.702* 

1.00 

4.  LF  SA 

.430* 

.430* 

.599* 

1.00 

5.LFTP 

.428* 

.527* 

.560* 

.603* 

1.00 

6.  Analogies 

.617* 

.753* 

.705* 

.576* 

.511* 

1.00 

7.  Scan 

.377* 

.410* 

.496* 

.422* 

.550* 

.465* 

1.00 

8.  AT  Eff. 

.514* 

.479* 

.625* 

.691* 

.549* 

.522* 

.422* 

1.00 

9.  AT  Safe 

.146 

.107 

.198 

.415* 

.169 

.143 

.005 

.368* 

1.00 

10.  AT  PA 

.413* 

.367* 

.379* 

.284* 

.340* 

.353* 

.375* 

.440* 

-.119 

1.00 

11.  AT  Total 

.490* 

.434* 

.562* 

.681* 

.495* 

.471* 

.353* 

.868* 

.686* 

.518* 

1.00 

12.  AT-SAT 

.727* 

.846* 

.913* 

.691* 

.699* 

.835* 

.633* 

.715* 

.246 

.477* 

.671* 

1.00 

Composite 

*  Correlation  is  significant  at  the  0.01  level  (2-tailed) 


Table  4.  Correlation  of  AT-SAT  Subtests 
With  Computer  Experience 


Test 

Computer  Experience 
Score 

Dials 

.196 

Angles 

.266** 

Amath 

.372** 

LF  SA 

447** 

LF  TP 

.283** 

Analogies 

.339** 

Scan 

.199 

AT  Eff. 

.365** 

AT  Safe 

.207* 

AT  PA 

.054 

AT  Total 

.318** 

AT-SAT 

.380** 

Composite 

**  Correlation  is  significant  at  the  0.01  level  (2-tailed) 
*  Correlation  is  significant  at  the  0.05  level  (2-tailed) 
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Table  5.  Hierarchical  Multiple  Regression  of  Age,  Education,  and  Computer  Experience 
on  Composite  AT-SAT  Score 


ATSAT 

Comp. 

Age 

Educ. 

B 

P 

AR2 

R2=.258* 

Age 

-.045 

-.196 

-.048 

.002 

Adjusted  R2 
=  .233 

Education 

.456** 

.129 

4.642 

.375 

.217** 

Computer 

Experience 

.380** 

.205* 

.389** 

.317 

.224 

.039* 

R=.508 

*p<.05 

**p<.01 


no  dear  pattern  of  relationship  emerged  between 
computer  experience  and  test  performance  based  on 
whether  the  test  was  static  or  dynamic. 

Hierarchical  regression  was  used  to  determine  the 
amount  of  variance  in  test  performance  that  was 
explained  by  computer  experience  after  age  and  edu¬ 
cation  were  taken  into  account.  The  AT-SAT  com¬ 
posite  test  score  and  each  AT-SAT  subtest  score  were 
used  as  dependent  variables  in  a  series  of  separate 
regression  analyses.  As  shown  in  Table  5,  age  did  not 
contribute  significantly  to  the  prediction  of  the  AT- 
SAT  composite  score.  When  entered  into  the  regres¬ 
sion  equation,  education  contributed  significantly  to 
the  prediction  of  AT-SAT  score  (AR2=.217.  pc.Ol). 
Computer  experience,  which  was  then  entered  into 
the  equation  after  education,  resulted  in  a  significant 
R2  change  in  predicting  the  AT-SAT  score  (AR2= . 039 , 
£<•05).  Based  on  the  adjusted  R2,  the  overall  model  of 
age,  education,  and  computer  experience  accounted  for 
23.3%  of  the  variance  in  the  composite  AT-SAT  score. 

A  summary  of  the  regression  analyses  for  each  AT- 
SAT  subtest  is  provided  in  Table  6.  For  each  analysis, 
the  independent  variables  were  entered  in  the  follow¬ 
ing  order:  age,  education,  and  computer  experience. 
A  separate  analysis  was  performed  for  each  AT-SAT 
subtest  score.  Age  did  not  contribute  significantly  to 
the  prediction  of  any  of  the  subtest  scores.  Education 
added  significantly  to  the  prediction  of  the  following 
scores:  Analogies  (AR2=-242,  pc.Ol),  Applied  Math 
(AR2=.218,  pc.Ol),  Angles  (AR2=-179,  pc.Ol),  LF 
Situational  Awareness  (AR2=.  148,  pc.0 1),  Dial  Read¬ 
ing  (AR2=.094,  pc.Ol),  AT  Total  (AR2=.09 1 ,  pc.Ol), 
AT  Efficiency  (AR2=.088.  pc.Ol),  AT  Safety 
(AR2=.083,  pc.Ol),  and  LF  Planning  and  Thinking 
Ahead  (&Rr=.Q55,  pc.Ol).  Once  age  and  education 
were  entered  into  the  regression  equation,  computer 


experience  added  significantly  only  to  the  prediction 
of  LF  Situational  Awareness  (AR2=.091,  pc.Ol),  AT 
Efficiency  (AR2=-064,  pc.05),  LF  Planning  and 
Thinking  Ahead  (AR2=.048,  pc.Ol),  and  AT  Total 
(AR2=.04l,  pc.05).  The  results  of  the  regression 
analysis  support  the  hypothesized  relationship  be¬ 
tween  computer  experience  and  performance  on  dy¬ 
namic  tests  requiring  use  of  a  mouse.  The  regression 
weights  and  adjusted  R2  for  each  separate  analysis  are 
listed  in  Table  6. 

Group  comparisons 

Differences  in  mean  computer  experience  scores 
based  on  gender,  race,  and  education  were  investi¬ 
gated  using  t-tests.  The  results,  summarized  in  Table 
7 >  revealed  no  significant  gender  or  racial  differences 
in  mean  computer  experience  score.  However,  par¬ 
ticipants  who  either  attended  or  graduated  from 
college  had  a  significantly  higher  computer  experi¬ 
ence  score  than  those  with  a  high  school  or  trade 
school  education,  t=-3.28  (92),  pc.05. 

A  2X2  between  subjects  MANOVA  was  performed 
on  the  ten  AT-SAT  subtests.  Education  and  self- 
reported  word-processing  experience  served  as  the 
independent  variables.  Order  of  entry  of  the  indepen¬ 
dent  variables  was  education,  then  word  processing 
experience.  Using  Wilks’  criterion,  there  was  a  sig¬ 
nificant  main  effect  of  both  education,  F(10, 
82)=2.75>  pc.Ol,  and  word  processing  experience, 
F(10,  82)  =2. 14,  pc.05.  The  interaction  was  not  sig¬ 
nificant.  The  results  reflected  a  moderate  association 
between  both  education  (r|2=.25)  and  word  process¬ 
ing  experience  (rp=.21)  and  the  combined  dependent 
variables  (DVs).  Differences  in  AT-SAT  test  scores 
associated  with  age  and  education  are  presented  in 
Appendix  B. 
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Table  6.  Regression  of  Age,  Education,  and  Computer  Experience  on 
AT-SAT  Sub-tests 


Dependent  Independent  p  AR2  Adjusted  R2 

Variable  Variables 


Dials 

Age  -.096  .005 

Education  .280  .094**  .073 

Computer  Experience  .068  .004 

Angles 

Age  -.070  .002 

Education  .382  .179**  .162 

Computer  Experience  .103  .008 

Amath 

Age  .019  .000 

Education  .374  .218**  .236 

Computer  Experience  .231  .042 

LFSA 

Age  -.053  .008 

Education  .244  .148**  .223 

Computer  Experience  .341  .091** 

LF  TP 

Age  .076  .002 

Education  .131  .055*  .074 

Computer  Experience  .247  .048* 

Analogies 

Age  -.091  .004 

Education  .432  .242**  .240 

Computer  Experience  .152  .018 

Scan 

Age  -.045  .006 

Education  .028  .011  .010 

Computer  Experience  .179  .025 

ATEff. 

Age  -.050  .007 

Education  .178  .088**  .131 

Computer  Experience  .285  .064* 

AT  Safe 

Age  -.153  .018 

Education  .259  .083**  .076 

Computer  Experience  .075  .004 

AT  PA 

Age  .184  .023 

Education  -.067  .000  .002 

Computer  Experience  .117  .011 

AT  Total 

Age  -.042  .004 

Education  .208  .091**  .107 

Computer  Experience  .228  .041* 

*p<.05 

**p<.01 


7 


Table  7.  Group  Means  and  Results  of  t-Test 
of  Computer  Experience 


Group 

Computer 
Experience  Score 

Mean  (SD) 

Gender 

•  Male 

39.75  (9.55) 

•  Female 

37.20  (9.77) 

Race 

•  Minority 

38.03  (8.15) 

•  Non-Minority 

39.00  (10.88) 

Education 

•  High  School 

34.19  (7.13) 

•  College 

40.83  (10.07)* 

*p<.05 


A  stepdown  analysis  was  performed  on  the  priori¬ 
tized  D  Vs  to  investigate  the  effect  of  each  main  effect 
on  the  individual  DVs.  The  D Vs  were  prioritized 
according  to  their  incremental  validity  in  predicting 
job  performance  (Ramos,  1999).  In  stepdown  analy¬ 
sis,  each  DV  was  analyzed  with  higher  priority  DVs 
treated  as  covariates  and  tested  as  a  univariate  ANOVA 
(Tabachnick&Fidell,  1996).  Results  of  the  stepd  own 
analysis  are  presented  in  Table  8.  Applied  math 
provided  a  unique  contribution  to  predicting  differ¬ 
ences  between  those  with  high  school/ trade  school  vs. 
college  education,  stepdown  F(l,  91)=12.75,  p<.01. 
Participants  who  attended  college  scored  higher  on 
the  applied  math  test  (M=  12.35)  than  did  those  with 
a  high  school  or  trade  school  education  (M=8.77). 
After  the  pattern  of  differences  measured  by  applied 
math  was  entered,  a  difference  was  also  found  on  the 
analogies  test,  stepdown  F(l,  90)=4.70,  £<.05.  The 
participants  who  attended  college  scored  higher  on 
the  analogies  test  (M=4.14)  than  did  those  partici¬ 
pants  with  a  high  school  or  trade  school  education 
(M=3.08).  Although  a  univariate  comparison  re¬ 
vealed  that  those  with  more  education  scored  higher 
on  the  angles  test,  univariate  F(l,  91)=9.10,  £<.01, 
LF  situational  awareness,  univariate  F(l,  91)=4.58, 
£<.05,  and  the  dial  reading  test,  univariate  F(l, 
9 1)=4.58,  £<.05,  these  differences  were  already  repre¬ 
sented  in  the  stepdown  analysis  by  higher-priority  DVs. 


As  shown  in  Table  8,  applied  math  made  a  unique 
contribution  to  predicting  differences  between  those 
with  and  those  without  word  processing  experience, 
stepdown  F(l,  91)=4.44,  £<.05.  Participants  with 
experience  using  a  computer  for  word  processing 
scored  higher  on  the  applied  math  test  (M=  11.62) 
than  did  those  with  no  word  processing  experience 
(M=9.49).  With  differences  due  to  the  applied  math 
score  already  entered,  the  analogies  test  score  contrib¬ 
uted  uniquely,  stepdown  F(l,  90)=8.14,  £<.01.  Par¬ 
ticipants  with  experience  using  a  computer  for  word 
processing  scored  higher  on  the  analogies  test 
(M=4.09)  than  did  those  with  no  word  processing 
experience  (M=3.12).  Although  a  univariate  com¬ 
parison  revealed  that  participants  with  word  process¬ 
ing  experience  scored  higher  on  LF  situational 
awareness,  univariate  F(l,  91)=1 1.01,  £<.01,  LF 
thinking  and  planning  ahead,  univariate  F(l, 
91)=6.76,  £<.05,  and  AT  efficiency,  univariate  F(l, 
91)=7.49>  £<.01,  these  differences  were  already  rep¬ 
resented  in  the  stepdown  analysis  by  applied  math 
and  analogies  scores. 

DISCUSSION 

The  results  of  the  analyses  described  above  provide 
evidence  of  a  relationship  between  prior  computer 
experience  and  performance  on  a  computerized  per¬ 
sonnel  selection  test.  In  general,  these  results  are 
consistent  with  those  reported  by  Keenan  (1999) 
during  the  AT-SAT  validation  study.  The  results  of 
the  Pearson  product-moment  correlations  reported 
in  the  current  study  provide  clear  evidence  of  a 
positive  relationship  between  computer  experience 
and  the  AT-SAT  composite  score.  Examination  of 
the  correlations  between  computer  experience  and 
the  individual  AT-SAT  subtest  scores  provides  in¬ 
sight  into  which  of  the  tests  are  more  affected. 

Only  three  of  the  subtest  scores  (dial  reading,  scan, 
and  AT  procedural  accuracy)  were  not  correlated 
with  computer  experience.  The  lack  of  a  relationship 
between  dial  reading  score  and  computer  experience 
is  not  surprising  given  that  the  dial  reading  test  is  a 
static  “page-turner”  multiple  choice  test.  The  exam¬ 
inee  need  only  use  the  keyboard  to  select  a  particular 
response  item.  Since  the  scan  and  AT  procedural 
accuracy  scores  are  based  on  dynamic  tests,  the  lack  of 
a  relationship  between  performance  and  computer 
experience  is  more  surprising.  However,  closer  in¬ 
spection  of  the  manner  in  which  these  scores  are 
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Table  8.  MANOVA  of  Education,  Word  Processing  Experience,  and  Their  Interaction 


IV 

DV 

Univariate 

Df 

Step-down 

Df 

F 

F 

Education 

Amath 

12.57** 

1/91 

12.57** 

1/91 

Analogies 

15.70** 

1/91 

4.70* 

1/90 

Angles 

9.10** 

1/91 

.03 

1/89 

LFSA 

4.58* 

1/91 

.01 

1/88 

LFTP 

1.39 

1/91 

.98 

1/87 

AT  Safe 

2.73 

1/91 

1.88 

1/86 

ATEff. 

2.88 

1/91 

.40 

1/85 

AT  PA 

.09 

1/91 

3.30 

1/84 

Dials 

4.58* 

1/91 

.00 

1/83 

Scan 

.05 

1/91 

2.71 

1/82 

Word  Processing  Experience 

Amath 

4.44* 

1/91 

4.44* 

1/91 

Analogies 

13.08** 

1/91 

8.14** 

1/90 

Angles 

3.78 

1/91 

.57 

1/89 

LFSA 

11.01** 

1/91 

3.52 

1/88 

LFTP 

6.76* 

1/91 

.45 

1/87 

AT  Safe 

.92 

1/91 

.06 

1/86 

ATEff. 

7  49** 

1/91 

.31 

1/85 

AT  PA  j 

.26 

1/91 

2.08 

1/84 

Dials 

1.02 

1/91 

1.28 

1/83 

Scan 

2.89 

1/91 

.22 

1/82 

Education  and  Word 

Amath 

7.00* 

1/91 

7.00* 

1/91 

Processing 

Analogies 

3.33 

1/91 

.11 

1/90 

Angles 

.98 

1/91 

1.30 

1/89 

LFSA 

4.27* 

1/91 

.51 

1/88 

LFTP 

1.28 

1/91 

.11 

1/87 

AT  Safe 

.65 

1/91 

.08 

1/86 

AT  Eff. 

2.20 

1/91 

.11 

1/85 

AT  PA 

.03 

1/91 

.91 

1/84 

Dials 

2.29 

1/91 

.29 

1/83 

Scan 

.04 

1/91 

3.38 

1/82 

*p<.05 

**p<.01 
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derived  provides  some  insight  into  this  issue.  The 
scan  test  requires  the  use  of  the  keyboard  for  input  of 
responses;  the  mouse  is  not  used  for  this  test.  The  AT 
procedural  accuracy  score  is  based  on  whether  or  not 
the  examinee  follows  the  correct  procedures  when 
directing  the  traffic  in  each  scenario.  For  example,  all 
planes  must  land  at  the  airport,  at  the  slowest  speed 
and  the  lowest  altitude.  These  rules  remain  the  same 
regardless  of  the  scenario.  Quick  and  efficient  mouse 
movement  is  not  required  for  an  examinee  to  score 
well  on  AT  procedural  accuracy. 

Contrary  to  what  was  hypothesized,  performance 
on  the  applied  math  and  angles  tests  (which  are  static 
multiple  choice  tests)  was  correlated  with  computer 
experience.  Further  investigation  revealed  that  this 
relationship  might  be  due  to  the  educational  level  of 
the  examinee.  The  results  of  t-tests  indicate  that 
people  with  more  education  also  had  more  computer 
experience.  Hierarchical  multiple  regression  revealed 
that  computer  experience  did  not  add  anything  be¬ 
yond  education  in  predicting  applied  math  or  angles 
scores.  Consequently,  it  was  concluded  that  the  sig¬ 
nificant  correlation  between  computer  experience 
and  performance  on  these  static  tests  was  due  to 
examinee  education  level. 

Computer  experience  added  significantly  to  the 
prediction  of  the  dynamic,  scenario-based  tests  that 
require  the  use  of  a  mouse.  After  controlling  for 
education,  computer  experience  added  to  the  predic¬ 
tion  of  LF  situational  awareness  score,  LF  planning 
and  thinking  ahead  score,  and  AT  efficiency  score. 
These  results  demonstrate  that,  after  controlling  for 
education,  computer  experience  was  related  only  to 
performance  on  dynamic  scenario-based  tests  that 
require  use  of  a  mouse.  The  LF  and  ATST  items 
require  fast  and  efficient  mouse  manipulation.  The 
relationship  between  computer  experience  and  per¬ 
formance  on  these  subtests  was  enough  to  influence 
the  relationship  between  computer  experience  and 
performance  on  the  overall  AT-SAT  composite  score. 
The  hierarchical  multiple  regression  with  AT-SAT 
composite  as  the  DV  revealed  that  computer  experi¬ 
ence  produced  a  significant  change  in  R2  beyond 
education.  Consequently,  people  with  more  com¬ 
puter  experience  received  higher  composite  AT-SAT 
scores.  Whether  or  not  this  relationship  is  due  to 
overall  experience  with  a  computer  or  to  experience 
with  a  mouse  in  particular  is  not  known;  the  com¬ 
puter  experience  questionnaire  did  not  contain  a 


question  specific  to  the  use  of  a  mouse.  Such  a 
question  should  be  included  in  the  future  as  it  has 
implications  for  training  interventions. 

The  results  of  the  MANOVA  stepdown  analysis 
suggest  that  the  pattern  of  differences  on  AT-SAT 
subtests  between  people  with  high  school/  trade  school 
education  and  college  education  were  best  measured 
by  scores  on  applied  math  and  analogies.  The  same  is 
true  of  the  differences  between  people  with  computer 
word  processing  experience  vs.  those  without  com¬ 
puter  word  processing  experience.  Consequently,  the 
influence  of  the  tests  that  provide  an  advantage  to 
those  with  computer  experience  (LF,  ATST)  is  mini¬ 
mal  when  the  subtest  scores  are  combined  in  a  linear 
fashion  based  on  their  incremental  validity  in  pre¬ 
dicting  job  performance. 

In  summary,  the  results  revealed  that  people  with 
a  higher  level  of  education  also  had  more  computer 
experience.  Once  education  level  was  controlled  for, 
computer  experience  was  correlated  with  performance 
on  dynamic  tests  that  required  the  use  of  a  mouse. 
This  relationship  was  sufficient  to  influence  the  se¬ 
lection  test  composite  score,  as  demonstrated  by  the 
finding  that  computer  experience  added  to  predic¬ 
tion  of  the  AT-SAT  composite  score  even  after  con¬ 
trolling  for  education.  The  influence  of  these  dynamic 
tests  on  the  composite  score  might  be  a  function  of 
the  weighting  of  these  tests  in  the  composite’s  calcu¬ 
lation.  Education  was  found  to  be  most  predictive  of 
performance  on  AT-SAT.  Currently,  any  adverse 
impact  associated  with  the  AT-SAT  battery  is  un¬ 
known  since  the  test  is  not  yet  operational.  However, 
people  with  less  computer  experience  may  be  at  a 
disadvantage  when  taking  a  computerized  test  that 
requires  the  use  of  a  mouse  to  complete  a  dynamic 
test.  Since  people  of  all  races  do  not  have  equal  access 
to  computers  (U.S.  Department  of  Commerce,  1999), 
this  disadvantage  may  contribute  to  adverse  impact. 
The  potential  for  disadvantage  will  depend  on  the 
number  of  such  tests  included  in  a  selection  battery, 
as  well  as  their  weight  in  calculating  the  composite 
score. 

The  relationship  between  computer  experience 
and  performance  on  a  computerized  selection  test  has 
important  implications  for  personnel  decision  mak¬ 
ing.  If  dynamic  scenario-based  tests  requiring  the  use 
of  a  mouse  are  heavily  weighted  in  the  calculation  of 
a  selection  test  score,  the  ranking  of  examinees  among 
qualified  applicants  may  be  affected  by  whether  or 


10 


not  they  had  previously  used  a  computer.  Future 
studies  should  explore  this  issue  as  well  as  the  extent 
to  which  computer  experience  adds  incremental  va¬ 
lidity  over  a  selection  test  in  predicting  job  perfor¬ 
mance.  Items  pertaining  to  mouse  and  video  game 
experience  should  also  be  added  to  the  computer 
experience  questionnaire,  as  these  may  help  isolate 
the  specific  types  of  training  that  are  most  likely  to  be 
effective  in  reducing  disparity  between  those  with 
and  without  prior  computer  experience.  This  can 
assist  in  the  development  of  a  computer  training 
intervention.  It  may  be  that  such  training  need  only 
focus  on  mouse  usage,  rather  than  on  such  things  as 
operating  systems  and  databases.  Future  research 
should  also  investigate  the  extent  to  which  such 
training  may  change  the  relationship  between  com¬ 
puter  experience  and  test  performance  so  that  the 
impact  on  personnel  decision-making  is  minimized. 
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APPENDIX  A 


Computer  Use  and  Experience  Questionnaire 

1 .  I  frequently  read  computer  magazines  or  other  sources  of  information  that  describe  new  computer 
technology. 

2.  I  know  how  to  recover  deleted  or  “lost  data”  on  a  computer  or  PC. 

3.  I  know  what  LAN  is. 

4.  I  know  what  an  operating  system  is. 

5.  I  know  how  to  write  computer  programs. 

6.  I  know  how  to  install  software  on  a  personal  computer. 

7.  I  know  what  e-mail  is. 

8.  I  know  what  a  database  is. 

9.  I  am  computer  literate. 

10. 1  regularly  use  a  PC  for  word  processing. 

1 1.  I  often  use  a  mainframe  computer  system. 

12.  I  am  good  at  using  computers. 


A1 


APPENDIX  B 


Mean  AT-SAT  Scores  by  Age  and  Education 


AT-SAT  Score 

18-21 

Age 

22-26 

27-30 

F 

Education 

H.S.  or  Trade  College 

F 

Amath 

12.15 

12.09 

10.87 

.54 

8.82 

13.14 

17.0 

(5.03) 

(5.31) 

(5.45) 

(4.61) 

(4.93) 

* 

Analogies 

4.07 

4.01 

3.68 

.59 

3.04 

4.38 

22.2 

(1.26) 

(1.51) 

(1.56) 

(1.28) 

(1.33) 

* 

Angles 

11.65 

11.20 

10.83 

.40 

9.53 

12.05 

13.2 

(3.33) 

(3.51) 

(3.38) 

(3.70) 

(2.92) 

* 

LFSA 

1.87 

1.72 

1.52 

1.5 

1.39 

1.87 

0° 

bo 

* 

(.81) 

(.76) 

(.71) 

(.61) 

(.79) 

LF  TP 

3.71 

3.75 

3.65 

.06 

3.41 

3.86 

3.7* 

(1.15) 

(1.11) 

(i.oi) 

(1.04) 

(.18) 

AT  Safe 

2.89 

2.96 

2.58 

1.9 

2.60 

2.95 

4.1* 

(.89) 

(.80) 

(.70) 

(.92) 

(.73) 

ATEff. 

2.70 

2.72 

2.39 

1.8 

2.35 

2.75 

6.0* 

(.75) 

(.88) 

(.55) 

(.68) 

(.78) 

AT  PA 

1.85 

1.87 

2.05 

1.3 

1.93 

1.91 

.03 

(.59) 

(.51) 

(.49) 

(.60) 

(.50) 

Dials 

9.92 

9.91 

9.48 

.51 

9.10 

10.13 

6.6* 

(1.36) 

(1.69) 

(2.57) 

(2.08) 

(1.71) 

Scan 

8.10 

8.45 

8.31 

1.1 

8.40 

8.67 

.45 

(1-69) 

(1.99) 

(1.79) 

(1.80) 

(1.88) 

*p<.05 

Standard  Deviation  in  (  ). 
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